07/04/2024 - 13/04/2024

10/04/2024 14:41

When starting the frontends, I noticed there was an ipmi communication failure for slot 1.

So I disabled slot 1 in the ODB and was able to start the frontends. Then I was able to start a run for ~20 minutes:

14:38:06.738 2024/04/10 [mhttpd,INFO] Run #65 stopped

14:38:00.523 2024/04/10 [MasterGM2,INFO] End of Run: DC7 Triggers Received 101970  Count triggers 101970

14:37:48.015 2024/04/10 [MasterGM2,TALK] Alarm: CCC Run Aborted

14:36:48.445 2024/04/10 [MasterGM2,TALK] Alarm: CCC Run Aborted

14:19:18.733 2024/04/10 [mhttpd,INFO] Run #65 started

14:38:06.738 2024/04/10 [mhttpd,INFO] Run #65 stopped

14:38:00.523 2024/04/10 [MasterGM2,INFO] End of Run: DC7 Triggers Received 101970  Count triggers 101970

14:37:48.015 2024/04/10 [MasterGM2,TALK] Alarm: CCC Run Aborted

14:36:48.445 2024/04/10 [MasterGM2,TALK] Alarm: CCC Run Aborted

14:19:18.733 2024/04/10 [mhttpd,INFO] Run #65 started

I also noticed:

tcp_client_eor(505): end-of-run TCP fill number 101970
--> entering setMasterRegister to write node CBUF.ACQUIRE
...
--> entering setMasterRegister to write node CBUF.ACQUIRE
ReadXBytes(919): socket file descriptor 48, request 8 bytes, read 0 bytes, tries 1000
10-04-24 14:38:04.625942 [7efec77fe700] ERROR - Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.4:50001
Caught Exception
setMasterRegister(799): uHAL Exception: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.4:50001

10-04-24 14:38:06.626302 [7efec77fe700] ERROR - Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.4:50001
Caught Exception
setMasterRegister(799): uHAL Exception: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.4:50001

<-- leaving getMasterRegister after failure
...
<-- leaving getMasterRegister after success

tcp_client_eor(505): end-of-run TCP fill number 101970
--> entering setMasterRegister to write node CBUF.ACQUIRE
...
--> entering setMasterRegister to write node CBUF.ACQUIRE
ReadXBytes(919): socket file descriptor 48, request 8 bytes, read 0 bytes, tries 1000
10-04-24 14:38:04.625942 [7efec77fe700] ERROR - Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.4:50001
Caught Exception
setMasterRegister(799): uHAL Exception: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.4:50001

10-04-24 14:38:06.626302 [7efec77fe700] ERROR - Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.4:50001
Caught Exception
setMasterRegister(799): uHAL Exception: Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.4:50001

<-- leaving getMasterRegister after failure
...
<-- leaving getMasterRegister after success

After this, I was unable to ping 192.168.1.4 or read its status with python3 read_status.py 1 4. This indicates to me the WFD5 shut off for some reason (overheating?)